{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lab 3 - Network Intrusion Detection\n",
    "- Daniel Vance\n",
    "- Peter Thung\n",
    "- Ravi Srinivas\n",
    "- Randy Balzer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Business Understanding"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Describe the purpose of the data set you selected \n",
    "- *Why was this data collected in the first place?* - The data set selected is the \"UNSW_NB15\" network traffic data set. The data set was created to evaluate Network Intrusion Detection Systems (NIDS). According to the creators of the data set, N. Moustafa and J. Slay from the Australian Defence Force Academy, the \"quality of a NIDS data set reflects two important characteristics: comprehensive reflection of contemporary threats and an inclusive normal range of traffic [1].\" Before this data set was generated, they argued that existing data sets used to train and test NIDS were not representative of current traffic flows and threats[2].\n",
    "- Describe how you would define and measure the outcomes from the dataset. That is: \n",
    "    - *Why is this data important?* - The importance of this data set is that it will allow NIDS to be evaluated better, which will increase their performance, increase their protective power and reduce the chance of false positives and false negatives.\n",
    "    - *How do you know if you have mined useful knowledge from the dataset?* - We will know we mined useful knowledge from the data set if we can determine, based on a collection of packets of traffic data, whether the features of those packets indicate an attack or just normal traffic.\n",
    "    - *How would you measure the effectiveness of a good prediction algorithm? Be specific.*\n",
    "    \n",
    "With our particular dataset there are a couple of different view points on which performance metrics a model should be tuned for for this particular dataset:\n",
    "\n",
    "1) False negatives are probably worse than false positives. If a malicious packet gets through undetected, it could potentially do bad things without being noticed, whereas a false positive could be rescreened or reviewed to clear the packet. Note: however, if the system has protection in depth, perhaps down stream systems may catch the packet or some effect of the malicous abnormal packet (e.g. host based intrusion detection system, so it may not be that bad). Recall may be a good metric to gauge model performance based on highlighting false negatives. Recall is defined as the number of True Postivies divided by the number of True Positives and the number of False Negatives, also known as Sensitivity or the True Postive Rate. \n",
    "\n",
    "2) However, if there are too may false positives, where the system over classifies packets as malcious when they really aren't then it could be interfering with normal operations and impact business operations. On the assumption that it takes some signficant amount of time to clear a tagged (false) positive packet. If a tool identifies too many false positives, people would lose trust with the system. Based on this view, precision which is defined as the number of True Positives divided by the number of True Positives and False positives would be a good metric to use to gauge performance of models used for predicting abnormal packets.\n",
    "\n",
    "3) Based on these view points, a metric which combines both would be the F1 Score which is 2*((precision*recall)/(precision+recall)) also know as the F Score or F Measure which tries to convey the balance between precision and recall. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Source*\n",
    "1. Moustafa, Nour, and Jill Slay. \"UNSW-NB15: a comprehensive data set for network intrusion detection systems (UNSW-NB15 network data set).\"Military Communications and Information Systems Conference (MilCIS), 2015. IEEE, 2015.\n",
    "2. Moustafa, Nour, and Jill Slay. \"The evaluation of Network Anomaly Detection Systems: Statistical analysis of the UNSW-NB15 data set and the comparison with the KDD99 data set.\" Information Security Journal: A Global Perspective (2016): 1-14.\n",
    "3. https://www.unsw.adfa.edu.au/australian-centre-for-cyber-security/cybersecurity/ADFA-NB15-Datasets/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Understanding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Imports\n",
    "from matplotlib import pyplot as plt\n",
    "from sklearn.model_selection import cross_val_score\n",
    "from sklearn.cross_validation import ShuffleSplit\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.cross_validation import StratifiedKFold\n",
    "from sklearn.grid_search import GridSearchCV\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.metrics import roc_curve, auc\n",
    "from sklearn import metrics as mt\n",
    "from sklearn.svm import SVC\n",
    "\n",
    "from sklearn.pipeline import FeatureUnion\n",
    "from sklearn.feature_selection import SelectKBest\n",
    "\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.model_selection import StratifiedKFold\n",
    "\n",
    "from sklearn import cross_validation\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "\n",
    "from sklearn.naive_bayes import GaussianNB\n",
    "\n",
    "from sklearn.decomposition import RandomizedPCA \n",
    "from sklearn.decomposition import PCA\n",
    "from sklearn import preprocessing\n",
    "from sklearn.discriminant_analysis import LinearDiscriminantAnalysis as LDA\n",
    "\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn import metrics as mt\n",
    "\n",
    "import seaborn as sns\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "import matplotlib.colors as colors\n",
    "\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.cluster import Birch, MiniBatchKMeans\n",
    "from sklearn.datasets.samples_generator import make_blobs\n",
    "\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Load UNSW_NB15 into a Pandas dataframe\n",
    "# df = pd.read_csv('https://github.com/danielbots/network_intrusion_detection_system/blob/master/UNSW_NB15_training_set.csv') # read in the csv file\n",
    "df = pd.read_csv('UNSW_NB15_training_set.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>dur</th>\n",
       "      <th>spkts</th>\n",
       "      <th>dpkts</th>\n",
       "      <th>sbytes</th>\n",
       "      <th>dbytes</th>\n",
       "      <th>rate</th>\n",
       "      <th>sttl</th>\n",
       "      <th>dttl</th>\n",
       "      <th>sload</th>\n",
       "      <th>...</th>\n",
       "      <th>ct_src_dport_ltm</th>\n",
       "      <th>ct_dst_sport_ltm</th>\n",
       "      <th>ct_dst_src_ltm</th>\n",
       "      <th>is_ftp_login</th>\n",
       "      <th>ct_ftp_cmd</th>\n",
       "      <th>ct_flw_http_mthd</th>\n",
       "      <th>ct_src_ltm</th>\n",
       "      <th>ct_srv_dst</th>\n",
       "      <th>is_sm_ips_ports</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>82332.000000</td>\n",
       "      <td>82332.000000</td>\n",
       "      <td>82332.000000</td>\n",
       "      <td>82332.000000</td>\n",
       "      <td>8.233200e+04</td>\n",
       "      <td>8.233200e+04</td>\n",
       "      <td>8.233200e+04</td>\n",
       "      <td>82332.000000</td>\n",
       "      <td>82332.000000</td>\n",
       "      <td>8.233200e+04</td>\n",
       "      <td>...</td>\n",
       "      <td>82332.000000</td>\n",
       "      <td>82332.000000</td>\n",
       "      <td>82332.000000</td>\n",
       "      <td>82332.000000</td>\n",
       "      <td>82332.000000</td>\n",
       "      <td>82332.000000</td>\n",
       "      <td>82332.000000</td>\n",
       "      <td>82332.000000</td>\n",
       "      <td>82332.000000</td>\n",
       "      <td>82332.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>41166.500000</td>\n",
       "      <td>1.006756</td>\n",
       "      <td>18.666472</td>\n",
       "      <td>17.545936</td>\n",
       "      <td>7.993908e+03</td>\n",
       "      <td>1.323379e+04</td>\n",
       "      <td>8.241089e+04</td>\n",
       "      <td>180.967667</td>\n",
       "      <td>95.713003</td>\n",
       "      <td>6.454902e+07</td>\n",
       "      <td>...</td>\n",
       "      <td>4.928898</td>\n",
       "      <td>3.663011</td>\n",
       "      <td>7.456360</td>\n",
       "      <td>0.008284</td>\n",
       "      <td>0.008381</td>\n",
       "      <td>0.129743</td>\n",
       "      <td>6.468360</td>\n",
       "      <td>9.164262</td>\n",
       "      <td>0.011126</td>\n",
       "      <td>0.550600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>23767.345519</td>\n",
       "      <td>4.710444</td>\n",
       "      <td>133.916353</td>\n",
       "      <td>115.574086</td>\n",
       "      <td>1.716423e+05</td>\n",
       "      <td>1.514715e+05</td>\n",
       "      <td>1.486204e+05</td>\n",
       "      <td>101.513358</td>\n",
       "      <td>116.667722</td>\n",
       "      <td>1.798618e+08</td>\n",
       "      <td>...</td>\n",
       "      <td>8.389545</td>\n",
       "      <td>5.915386</td>\n",
       "      <td>11.415191</td>\n",
       "      <td>0.091171</td>\n",
       "      <td>0.092485</td>\n",
       "      <td>0.638683</td>\n",
       "      <td>8.543927</td>\n",
       "      <td>11.121413</td>\n",
       "      <td>0.104891</td>\n",
       "      <td>0.497436</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>2.400000e+01</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>...</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>20583.750000</td>\n",
       "      <td>0.000008</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.140000e+02</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>2.860611e+01</td>\n",
       "      <td>62.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.120247e+04</td>\n",
       "      <td>...</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>41166.500000</td>\n",
       "      <td>0.014138</td>\n",
       "      <td>6.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>5.340000e+02</td>\n",
       "      <td>1.780000e+02</td>\n",
       "      <td>2.650177e+03</td>\n",
       "      <td>254.000000</td>\n",
       "      <td>29.000000</td>\n",
       "      <td>5.770032e+05</td>\n",
       "      <td>...</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>61749.250000</td>\n",
       "      <td>0.719360</td>\n",
       "      <td>12.000000</td>\n",
       "      <td>10.000000</td>\n",
       "      <td>1.280000e+03</td>\n",
       "      <td>9.560000e+02</td>\n",
       "      <td>1.111111e+05</td>\n",
       "      <td>254.000000</td>\n",
       "      <td>252.000000</td>\n",
       "      <td>6.514286e+07</td>\n",
       "      <td>...</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>6.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>7.000000</td>\n",
       "      <td>11.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>82332.000000</td>\n",
       "      <td>59.999989</td>\n",
       "      <td>10646.000000</td>\n",
       "      <td>11018.000000</td>\n",
       "      <td>1.435577e+07</td>\n",
       "      <td>1.465753e+07</td>\n",
       "      <td>1.000000e+06</td>\n",
       "      <td>255.000000</td>\n",
       "      <td>253.000000</td>\n",
       "      <td>5.268000e+09</td>\n",
       "      <td>...</td>\n",
       "      <td>59.000000</td>\n",
       "      <td>38.000000</td>\n",
       "      <td>63.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>16.000000</td>\n",
       "      <td>60.000000</td>\n",
       "      <td>62.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>8 rows × 41 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                 id           dur         spkts         dpkts        sbytes  \\\n",
       "count  82332.000000  82332.000000  82332.000000  82332.000000  8.233200e+04   \n",
       "mean   41166.500000      1.006756     18.666472     17.545936  7.993908e+03   \n",
       "std    23767.345519      4.710444    133.916353    115.574086  1.716423e+05   \n",
       "min        1.000000      0.000000      1.000000      0.000000  2.400000e+01   \n",
       "25%    20583.750000      0.000008      2.000000      0.000000  1.140000e+02   \n",
       "50%    41166.500000      0.014138      6.000000      2.000000  5.340000e+02   \n",
       "75%    61749.250000      0.719360     12.000000     10.000000  1.280000e+03   \n",
       "max    82332.000000     59.999989  10646.000000  11018.000000  1.435577e+07   \n",
       "\n",
       "             dbytes          rate          sttl          dttl         sload  \\\n",
       "count  8.233200e+04  8.233200e+04  82332.000000  82332.000000  8.233200e+04   \n",
       "mean   1.323379e+04  8.241089e+04    180.967667     95.713003  6.454902e+07   \n",
       "std    1.514715e+05  1.486204e+05    101.513358    116.667722  1.798618e+08   \n",
       "min    0.000000e+00  0.000000e+00      0.000000      0.000000  0.000000e+00   \n",
       "25%    0.000000e+00  2.860611e+01     62.000000      0.000000  1.120247e+04   \n",
       "50%    1.780000e+02  2.650177e+03    254.000000     29.000000  5.770032e+05   \n",
       "75%    9.560000e+02  1.111111e+05    254.000000    252.000000  6.514286e+07   \n",
       "max    1.465753e+07  1.000000e+06    255.000000    253.000000  5.268000e+09   \n",
       "\n",
       "           ...       ct_src_dport_ltm  ct_dst_sport_ltm  ct_dst_src_ltm  \\\n",
       "count      ...           82332.000000      82332.000000    82332.000000   \n",
       "mean       ...               4.928898          3.663011        7.456360   \n",
       "std        ...               8.389545          5.915386       11.415191   \n",
       "min        ...               1.000000          1.000000        1.000000   \n",
       "25%        ...               1.000000          1.000000        1.000000   \n",
       "50%        ...               1.000000          1.000000        3.000000   \n",
       "75%        ...               4.000000          3.000000        6.000000   \n",
       "max        ...              59.000000         38.000000       63.000000   \n",
       "\n",
       "       is_ftp_login    ct_ftp_cmd  ct_flw_http_mthd    ct_src_ltm  \\\n",
       "count  82332.000000  82332.000000      82332.000000  82332.000000   \n",
       "mean       0.008284      0.008381          0.129743      6.468360   \n",
       "std        0.091171      0.092485          0.638683      8.543927   \n",
       "min        0.000000      0.000000          0.000000      1.000000   \n",
       "25%        0.000000      0.000000          0.000000      1.000000   \n",
       "50%        0.000000      0.000000          0.000000      3.000000   \n",
       "75%        0.000000      0.000000          0.000000      7.000000   \n",
       "max        2.000000      2.000000         16.000000     60.000000   \n",
       "\n",
       "         ct_srv_dst  is_sm_ips_ports         label  \n",
       "count  82332.000000     82332.000000  82332.000000  \n",
       "mean       9.164262         0.011126      0.550600  \n",
       "std       11.121413         0.104891      0.497436  \n",
       "min        1.000000         0.000000      0.000000  \n",
       "25%        2.000000         0.000000      0.000000  \n",
       "50%        5.000000         0.000000      1.000000  \n",
       "75%       11.000000         0.000000      1.000000  \n",
       "max       62.000000         1.000000      1.000000  \n",
       "\n",
       "[8 rows x 41 columns]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# df.describe() calculates the count, mean, standard deviation, min, max, and quartiles \n",
    "# of the data less the nominal (object) columns. \n",
    "df.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "proto:\n",
      "\n",
      "service:\n",
      "\n",
      "state:\n",
      "\n",
      "attack_cat:\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# List the unique values of each nominal column\n",
    "header = df.columns.values.tolist()\n",
    "for column in header:\n",
    "    if column in df.select_dtypes(include=[\"object\"]):\n",
    "        print (column + \":\\n\"), df[column].unique()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Verify data quality: Explain any missing values, duplicate data, and outliers.\n",
    "    - Are those mistakes? How do you deal with these problems? Be specific."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 82328 entries, 0 to 82331\n",
      "Data columns (total 45 columns):\n",
      "id                   82328 non-null int64\n",
      "dur                  82328 non-null float64\n",
      "proto                82328 non-null object\n",
      "service              82328 non-null object\n",
      "state                82328 non-null object\n",
      "spkts                82328 non-null int64\n",
      "dpkts                82328 non-null int64\n",
      "sbytes               82328 non-null int64\n",
      "dbytes               82328 non-null int64\n",
      "rate                 82328 non-null float64\n",
      "sttl                 82328 non-null int64\n",
      "dttl                 82328 non-null int64\n",
      "sload                82328 non-null float64\n",
      "dload                82328 non-null float64\n",
      "sloss                82328 non-null int64\n",
      "dloss                82328 non-null int64\n",
      "sinpkt               82328 non-null float64\n",
      "dinpkt               82328 non-null float64\n",
      "sjit                 82328 non-null float64\n",
      "djit                 82328 non-null float64\n",
      "swin                 82328 non-null int64\n",
      "stcpb                82328 non-null int64\n",
      "dtcpb                82328 non-null int64\n",
      "dwin                 82328 non-null int64\n",
      "tcprtt               82328 non-null float64\n",
      "synack               82328 non-null float64\n",
      "ackdat               82328 non-null float64\n",
      "smean                82328 non-null int64\n",
      "dmean                82328 non-null int64\n",
      "trans_depth          82328 non-null int64\n",
      "response_body_len    82328 non-null int64\n",
      "ct_srv_src           82328 non-null int64\n",
      "ct_state_ttl         82328 non-null int64\n",
      "ct_dst_ltm           82328 non-null int64\n",
      "ct_src_dport_ltm     82328 non-null int64\n",
      "ct_dst_sport_ltm     82328 non-null int64\n",
      "ct_dst_src_ltm       82328 non-null int64\n",
      "is_ftp_login         82328 non-null int64\n",
      "ct_ftp_cmd           82328 non-null int64\n",
      "ct_flw_http_mthd     82328 non-null int64\n",
      "ct_src_ltm           82328 non-null int64\n",
      "ct_srv_dst           82328 non-null int64\n",
      "is_sm_ips_ports      82328 non-null int64\n",
      "attack_cat           82328 non-null object\n",
      "label                82328 non-null int64\n",
      "dtypes: float64(11), int64(30), object(4)\n",
      "memory usage: 28.9+ MB\n"
     ]
    }
   ],
   "source": [
    "# Overwrite the existing dataframe with the new dataframe that does not contain the \n",
    "# four unwanted records and confirm we have 4 less records\n",
    "df = df[df.is_ftp_login != 2]\n",
    "df.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Outliers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of observations 82328\n",
      "Number of lower outliers 0\n",
      "Number of upper outliers 6201\n"
     ]
    }
   ],
   "source": [
    "# using += 1.5 IQR to determine outliers from continous/oridnal numbers. \n",
    "df_desc_table = df.describe() # result: 41 numeric types\n",
    "col_rate = df_desc_table['rate']\n",
    "df_sub2 =df[['rate']]\n",
    "\n",
    "q1 = df_desc_table.iloc[4]['rate']  # 4 = 25% row of description table or Q1\n",
    "q3 = df_desc_table.iloc[6]['rate']  # 6 = 75% row of description table or Q3\n",
    "iqr = q3 - q1\n",
    "upper_outlier_threshold = q3 + (1.5 * iqr) \n",
    "lower_outlier_threshold = q1 - (1.5 * iqr)\n",
    "\n",
    "col_index = df.columns\n",
    "col_index.get_values() # Returns an array of column names \n",
    "col = df['rate']\n",
    "lower_outliers = col[col < lower_outlier_threshold]\n",
    "upper_outliers = col[col > upper_outlier_threshold]\n",
    "\n",
    "print (\"Number of observations\", len(col))\n",
    "print (\"Number of lower outliers\", len(lower_outliers))\n",
    "print (\"Number of upper outliers\", len(upper_outliers))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Column name: #obserations: #lower_outliers: #upper_outliers: Max Values\n",
      "dur: 82328 : 0 : 5868 : 59.999989\n",
      "spkts: 82328 : 0 : 10196 : 10646.0\n",
      "dpkts: 82328 : 0 : 8907 : 11018.0\n",
      "sbytes: 82328 : 0 : 9270 : 14355774.0\n",
      "dbytes: 82328 : 0 : 12308 : 14657531.0\n",
      "rate: 82328 : 0 : 6201 : 1000000.003\n",
      "sload: 82328 : 0 : 6715 : 5268000256.0\n",
      "dload: 82328 : 0 : 18112 : 20821108.0\n",
      "sloss: 82328 : 0 : 5499 : 5319.0\n",
      "dloss: 82328 : 0 : 11272 : 5507.0\n",
      "sinpkt: 82328 : 0 : 5667 : 60009.992\n",
      "dinpkt: 82328 : 0 : 4717 : 57739.24\n",
      "sjit: 82328 : 0 : 6320 : 1483830.917\n",
      "djit: 82328 : 0 : 8569 : 463199.2401\n",
      "tcprtt: 82328 : 0 : 2019 : 3.821465\n",
      "synack: 82328 : 0 : 2954 : 3.226788\n",
      "ackdat: 82328 : 0 : 2480 : 2.928778\n",
      "smean: 82328 : 0 : 11928 : 1504.0\n",
      "dmean: 82328 : 0 : 9902 : 1500.0\n",
      "trans_depth: 82328 : 0 : 7582 : 131.0\n",
      "response_body_len: 82328 : 0 : 5657 : 5242880.0\n",
      "ct_srv_src: 82328 : 0 : 10093 : 63.0\n",
      "ct_state_ttl: 82328 : 0 : 1833 : 6.0\n",
      "ct_dst_ltm: 82328 : 0 : 10479 : 59.0\n",
      "ct_src_dport_ltm: 82328 : 0 : 11476 : 59.0\n",
      "ct_dst_sport_ltm: 82328 : 0 : 10907 : 38.0\n",
      "ct_dst_src_ltm: 82328 : 0 : 12789 : 63.0\n",
      "is_ftp_login: 82328 : 0 : 674 : 1.0\n",
      "ct_ftp_cmd: 82328 : 0 : 676 : 2.0\n",
      "ct_flw_http_mthd: 82328 : 0 : 7580 : 16.0\n",
      "ct_src_ltm: 82328 : 0 : 9579 : 60.0\n",
      "ct_srv_dst: 82328 : 0 : 10040 : 62.0\n",
      "is_sm_ips_ports: 82328 : 0 : 916 : 1.0\n",
      "Number of features with outliers: 33\n"
     ]
    }
   ],
   "source": [
    "# Code that calculates the number of outliers below and above for continous numeric features.\n",
    "# this is to help determine which particular feature to do box plots on.\n",
    "#ref: http://stackoverflow.com/questions/24901766/python-how-to-get-column-names-from-pandas-dataframe-but-only-for-continuous\n",
    "df_cont = df.select_dtypes(['float64', 'int64']) # a data frame of just continous variables\n",
    "### len(df_cont.columns)\n",
    "#df_cont.drop('id', axis=1, inplace=True) # 1 is the axis number ( 0 for rows 1 for columns), dropping id as it doesn't make sense to plot outliers for that column\n",
    "### len(df_cont.columns)\n",
    "print (\"Column name: #obserations: #lower_outliers: #upper_outliers: Max Values\")\n",
    "num_features_with_outliers = 0\n",
    "for col_name in df_cont.columns:\n",
    "    q1 = df_desc_table.iloc[4][col_name]  # 4 = 25% row of description table or Q1\n",
    "    q3 = df_desc_table.iloc[6][col_name]  # 6 = 75% row of description table or Q3\n",
    "    max_val = df_desc_table.iloc[7][col_name] # 7 = max value\n",
    "    iqr = q3 - q1\n",
    "    upper_outlier_threshold = q3 + (1.5 * iqr) \n",
    "    lower_outlier_threshold = q1 - (1.5 * iqr)\n",
    "    col = df[col_name]\n",
    "    lower_outliers = col[col < lower_outlier_threshold]\n",
    "    upper_outliers = col[col > upper_outlier_threshold]\n",
    "    if (len(lower_outliers) + len(upper_outliers)) > 0:\n",
    "        num_features_with_outliers+=1\n",
    "        print (col_name + \": \" + str(len(col)) + \" : \" + str(len(lower_outliers)) + \" : \" + str(len(upper_outliers)) + \" : \" + str(max_val))\n",
    "print (\"Number of features with outliers: \" + str(num_features_with_outliers))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cmap = sns.diverging_palette(220, 10, as_cmap=True) # one of the many color mappings\n",
    "\n",
    "#attack records\n",
    "df1_1 = df[['label', 'proto', 'state', 'dur', 'sbytes', 'dbytes', 'sttl', 'dttl', 'sloss', 'dloss']]\n",
    "df1_1 = df1_1[df1_1.label == 1]\n",
    "\n",
    "df1_2 = df[['label', 'service', 'sload', 'dload', 'spkts', 'dpkts', 'swin', 'dwin', 'stcpb', 'dtcpb']]\n",
    "df1_2 = df1_2[df1_2.label == 1]\n",
    "\n",
    "df1_3 = df[['label', 'smean', 'dmean', 'trans_depth', 'response_body_len', 'sjit', 'djit', 'tcprtt']]\n",
    "df1_3 = df1_3[df1_3.label == 1]\n",
    "\n",
    "## normal records\n",
    "df0_1 = df[['label', 'proto', 'state', 'dur', 'sbytes', 'dbytes', 'sttl', 'dttl', 'sloss', 'dloss']]\n",
    "df0_1 = df0_1[df0_1.label == 0]\n",
    "\n",
    "df0_2 = df[['label', 'service', 'sload', 'dload', 'spkts', 'dpkts', 'swin', 'dwin', 'stcpb', 'dtcpb']]\n",
    "df0_2 = df0_2[df0_2.label == 0]\n",
    "\n",
    "df0_3 = df[['label', 'smean', 'dmean', 'trans_depth', 'response_body_len', 'sjit', 'djit', 'tcprtt']]\n",
    "df0_3 = df0_3[df0_3.label == 0]\n",
    "\n",
    "\n",
    "## both attack and normal records\n",
    "df_1 = df[['label', 'proto', 'state', 'dur', 'sbytes', 'dbytes', 'sttl', 'dttl', 'sloss', 'dloss']]\n",
    "df_2 = df[['label', 'service', 'sload', 'dload', 'spkts', 'dpkts', 'swin', 'dwin', 'ct_src_dport_ltm', 'ct_dst_sport_ltm']]\n",
    "df_3 = df[['label', 'stcpb', 'dtcpb', 'smean', 'dmean', 'trans_depth', 'response_body_len', 'sjit', 'djit', 'tcprtt']]\n",
    "df_4 = df[['label', 'sinpkt', 'dinpkt', 'synack', 'ackdat', 'ct_srv_src', 'ct_state_ttl', 'ct_dst_ltm','ct_src_dport_ltm', 'ct_dst_sport_ltm']]\n",
    "df_5 = df[['label', 'ct_dst_src_ltm', 'is_ftp_login', 'ct_ftp_cmd', 'ct_flw_http_mthd', 'ct_src_ltm', 'ct_srv_dst', 'is_sm_ips_ports']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looking at the scatter plot matrix, there appears to be varying degrees of positive linear correlation between the following features: dbytes - dloss; dpkts - dloss; sbytes sloss; spkts - sloss; ct_srv_src - ct_dst_src_ltm; dpkts - dbytes Other features are more difficult to find signifiant linear correlation due to binary nature of variable. The heat map below will help identify them visually based on color gradient which relates to strength."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Modeling and Evaluation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Train and Adjust Parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "    Models:\n",
    "        K-Means - PETER\n",
    "        MiniBatch - RANDY\n",
    "        DBSCAN - RAVI\n",
    "        GMM - DANIEL\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Evaluate and Compare"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## MiniBatchKMeans\n",
    "\n",
    "Reference:\n",
    "http://scikit-learn.org/stable/modules/clustering.html#mini-batch-kmeans\n",
    "\n",
    "An optimized variant of the K-means clustering algorithm is the Mini Batch K-means algorithm which increases optimization by dividing the input data into mini-batches resulting in reduced computation time.  The tradeoff of converging to a local solution in each mini-batch is an overall reduction in the quality of the results, however, in practice the difference has been shown to be quite small.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Value error(No objects to concatenate): \n",
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 82328 entries, 0 to 82331\n",
      "Columns: 202 entries, id to attack_cat_Worms\n",
      "dtypes: float64(11), int64(30), uint8(161)\n",
      "memory usage: 39.0 MB\n"
     ]
    }
   ],
   "source": [
    "# One-hot encode our object features:\n",
    "\n",
    "# http://stackoverflow.com/questions/19482970/get-list-from-pandas-dataframe-column-headers\n",
    "# Surrounding code in try/except on case where there are no object type features to one-hot encode\n",
    "try:\n",
    "    tmp_df = df.describe(include=['O'])  # creates a temporary df with just categorical features that are of object type\n",
    "    categorical_object_col_name_list = tmp_df.columns.values.tolist()\n",
    "    for col_name in categorical_object_col_name_list:\n",
    "        #print col_name\n",
    "        tmp_df = pd.get_dummies(df[col_name], prefix=col_name)\n",
    "        df = pd.concat((df,tmp_df), axis=1)\n",
    "        df.drop(col_name, axis=1, inplace=True)  # go ahead and drop original feature as it has now been one-hot encoded\n",
    "except ValueError as e:\n",
    "    print (\"Value error({0}): \".format(e) )  # Note\n",
    "df.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 82328 entries, 0 to 82331\n",
      "Columns: 202 entries, id to attack_cat_Worms\n",
      "dtypes: float64(11), int64(30), uint8(161)\n",
      "memory usage: 39.0 MB\n",
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 82328 entries, 0 to 82331\n",
      "Columns: 201 entries, id to attack_cat_Worms\n",
      "dtypes: float64(11), int64(29), uint8(161)\n",
      "memory usage: 38.4 MB\n"
     ]
    }
   ],
   "source": [
    "# Go ahead and drop both 'label' \n",
    "dfcopy = df.copy(deep=True) # preserve original dataframe that has our dependent variable\n",
    "dfcopy.info()\n",
    "# we want to predict the X and y data as follows:\n",
    "if 'label' in dfcopy:\n",
    "    y = dfcopy['label'].values # get the labels we want\n",
    "    del dfcopy['label'] # get rid of the class label\n",
    "    X = dfcopy.values # use everything else to predict!\n",
    "dfcopy.info() # should have 190 entries\n",
    "    # X and y are now numpy matrices, by calling 'values' on the pandas data frames we\n",
    "    # have converted them into simple matrices to use with scikit learn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Comparison of cluster training times.\n",
      "\tkmeans train time: 1.04s\n",
      "inertia: 106110494664067739811840.000000\n",
      "\tmini batch train time: 0.42s\n",
      "inertia: 106484042969018074660864.000000\n",
      "\tdbscan train time: 28.68s\n",
      "Comparison of cluster training times using PCA for dimensionality reduction.\n",
      "\tkmeans train time: 0.06s\n",
      "inertia: 103703172876202615504896.000000\n",
      "\tmini batch train time: 0.19s\n",
      "inertia: 103858304635522665938944.000000\n",
      "\tdbscan train time: 3.88s\n"
     ]
    }
   ],
   "source": [
    "import time\n",
    "from sklearn.cluster import KMeans\n",
    "from sklearn.cluster import DBSCAN\n",
    "\n",
    "n_clusters = 2\n",
    "\n",
    "\n",
    "## PCA\n",
    "def pca(pc):\n",
    "    rpca=PCA(svd_solver='randomized', n_components=2)\n",
    "    return rpca.fit_transform(pc)\n",
    "\n",
    "\n",
    "## KMeans\n",
    "def kmeans(X):\n",
    "    k_means = KMeans(init='k-means++', n_clusters=n_clusters, n_init=1)\n",
    "    t0 = time.time()\n",
    "    k_means.fit(X)\n",
    "    t_kmeans = time.time() - t0\n",
    "    print (\"\\tkmeans train time: %.2fs\\n\\tinertia: %f\"%(t_kmeans, k_means.inertia_))\n",
    "    return()\n",
    "\n",
    "\n",
    "## MiniBatchKMeans\n",
    "def minibatchkmeans(X):\n",
    "    batch_size=100\n",
    "    mbk = MiniBatchKMeans( init='k-means++', n_clusters=2, batch_size=batch_size,\n",
    "                           n_init=10, max_no_improvement=10, verbose=0 )\n",
    "    t0 = time.time()\n",
    "    mbk.fit(X)\n",
    "    t_mini_batch = time.time() - t0\n",
    "    print (\"\\tmini batch train time: %.2fs\\n\\tinertia: %f\"%(t_mini_batch, mbk.inertia_))\n",
    "    return()\n",
    "\n",
    "\n",
    "## DBSCAN\n",
    "def dbscan(X):\n",
    "    \n",
    "    dbscan = DBSCAN(eps=0.15, min_samples=10)\n",
    "    t0 = time.time()\n",
    "    dbscan.fit(X)\n",
    "    t_dbscan = time.time() - t0\n",
    "    print (\"\\tdbscan train time: %.2fs\"%(t_dbscan))\n",
    "    \n",
    "    return()\n",
    "\n",
    "\n",
    "\n",
    "def main():    \n",
    "    \n",
    "    print(\"Comparison of cluster training times.\")\n",
    "    kmeans(X)\n",
    "    minibatchkmeans(X)\n",
    "    dbscan(X)\n",
    "    \n",
    "    pc = pca(X)\n",
    "    print(\"Comparison of cluster training times using PCA for dimensionality reduction.\")\n",
    "    kmeans(pc)\n",
    "    minibatchkmeans(pc)\n",
    "    dbscan(pc)\n",
    "    \n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "    main()\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Visualize Results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "    Can we visualize a data set with this many features?\n",
    "    Can we visualize on the attack categories?\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Summarize the Ramifications"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deployment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this phase, we understand that our evaluation model has met business objectives and is receiving reliable score data. Making the model exceptionally useful for businesses & organizations seeking to protect assets from potential cyberattacks. The model’s performance and desirability rests on some identified variables being, the overall performance of the model to protect from attacks, the costs associated with attacks landed/ missed, and also “how well does the model grow?” These factors could help a customer’s decision making process to employing an algorithmic method. Ultimately our job would be to efficiently and effectively scale the methodology appropriately for each customer. Interested customers would have a clear understanding of the nature of their business and the amount of protection needed for their size of an organization. To properly deploy a model to the customer, they would need to be aware of the overhead related in performing certain algorithms, mostly being time and money. The updating process of such a model could be continuous, either by streaming in live time by the seconds or delivered in blocks of time, such as hourly/daily/weekly.\n",
    "\n",
    "The customer would also have to make a decision on whether they wanted real time cyber attack protection or log scans. Real time coverage has overhead to consider, such as latency that the algorithm will cause, especially in high throughput networks. Log anaysis is much easier to implement, and if ran regularly enough, it may give the customers the reassurance they need that they have not been breached."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- *Be critical of your performance and tell the reader how you current model might be usable by other parties. Did you achieve your goals? If not, can you reign in the utility of your modeling?*\n",
    "- *How useful is your model for interested parties (i.e., the companies or organizations that might want to use it)?*\n",
    "- *How would your deploy your model for interested parties?*\n",
    "- *What other data should be collected?*\n",
    "- *How often would the model need to be updated, etc.?*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exceptional Work"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [conda root]",
   "language": "python",
   "name": "conda-root-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
